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ABSTRACT 



This study examines rankings of four-year liberal arts 
colleges by "U.S. News and World Report," focusing on whether institutional 
improvement efforts are likely to result in higher rankings. The paper 
explores how often changes in rankings occur; whether they tend to improve; 
and whether positive movements are as likely as negative ones. The study 
examined the 162 ranked colleges and their tier assignments to identify 
cross-tier movements over four time periods (1996-97, 1997-98, 1998-99, 
1999-2000) . Year-to-year comparisons showed no statistically significant 
change, and the study concludes that once included among the "best liberal 
arts colleges,” an institution is likely to stay there. Over the study 
period, 104 institutions showed no movement at all; what movement there was 
is characterized as "zero sum, " with the upward movement of some balanced by 
the downward movement of others. The paper concludes with two caveats: First, 
that it is important for institutions to determine their approximate rank and 
to decide whether any efforts to improve are likely to prove fruitful; and 
second, in the absence of ratings, would institutions continue to expend the 
same amount of effort on institutional improvement. (CH) 
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Background 



Each year the U. S. News and World Report publishes rankings of colleges and universities in 
the United States. As a guide to college applicants and their parents, these rankings are taken 
quite seriously. Decisions affecting innumerable academic careers have been and will 
continue to be influenced through these annual publications. Since colleges and universities 
recognize the impact that these published rankings (and many others) have, the leaders of 
many of these institutions are eager to improve their standing. Thus, presidents, admissions 
officers and others pay particularly close attention to the rankings and the criteria that 
influence them, undoubtedly aspiring to influence their own institutions in a way that will 
have a positive impact on the ratings and the benefits that will follow. 

Some would argue that such efforts should not be equated with direct improvements in 
institutions. Others might argue that there are overall benefits that stem from the efforts of 
U. S. News and World Report, and dozens of others, to perform a service for consumers. This 
paper does not deal directly with the issue of whether the time spent paying attention to 
college rankings is well spent in the sense that they lead directly to institutional 
improvements. However, the paper deals with the related issue of whether such efforts are 
well spent in the sense that they are likely to pay off in reaching the immediate goal of higher 
rankings. To state the issue another way, we asked the following questions: How often do 
changes in rankings occur? Do they frequently tend to improve? Are positive movements 
in rankings as likely as negative movements? This study is not intended to discourage those 
efforts but rather to provide an accurate picture of how much movement whether it is rising 
or falling movement has occurred historically. Such information is the best guide on how 
much movement, positive or negative, to expect in the future. It was hoped that the answers 
to these and related questions would help institutions to know whether their efforts to 
improve their rankings are likely to succeed. 



Method 



This report summarizes the results of a study of the rankings for the past 5 years in the U.S. 
News and World Report for nationally ranked four-year liberal arts colleges. Five years of 
rankings gave the possibility of movement over four years: 1996-97, 1997-98, 1998-99, and 
1999-2000. Colleges could stay the same over all four years or change. If they changed, they 
could change up or down repeatedly in several combinations. 

The method used involved entering the names of all colleges and their tier assignments (1-4) 
for each year into a spreadsheet. For tier one assignments only, we made entries of the 
unique institutional rank (1-40) which are published by U. S. News and World Report. 
Sorting the spreadsheet by college and by year within colleges afforded an opportunity to 
identify cross-tier movements. Most of the findings reported below stem from these simple 
steps and the various counts and other calculations that they made possible. 
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Results and Discussion 



If one looks at the probability of any changes taking place out of all opportunities for change 
to have occurred, that value turned out to be .14. In other words, change occurred in 14 
percent of the opportunities. Therefore most of the year to year comparisons showed no 
change. The probability of no change from one year to the next was .86-or 86 percent of 
the opportunities. Further, the probability of a change (.14) was evenly divided between 
rising and falling, with the probability of reaching a higher tier in the next being .07 and that 
of declining being also .07. 

Looking at the institutions, there were 162 represented over the 5 years. That is only slightly 
higher than the 160 that would be expected if there were 40 in each tier each year and there 
were no newcomers. In fact, there were several, but only very few, that appeared less than 5 
times. The breakdown was as follows: 

5 appearances: 158 (97.5%) 

4 appearances: 1 (0.6%) 

2 appearances: 1 (0.6%) 

1 appearance: 2 (1.2%) 

The obvious implication is that, once included among the "best liberal arts colleges," an 
institution is likely to stay there. The other side of this positive conclusion is that movement 
within the 160 or so best is extremely slow and difficult. There were 104 institutions that 
showed no movement at all over 5 appearances. This conservatism is shown most of all in 
Tier 1, which is partly due to the "ceiling" effect — Tier 1 institutions cannot rise to a higher 
tier. However, Tier 4 membership is also conservative, even though the institutions making 
up that tier are at most risk of falling off of the "best colleges" list completely. 

Since the bottom 3 tiers are not ranked, it is difficult to determine the average movement per 
institution over the 4 years of possible change. Such study fell outside the resources of time 
available for the current study. Fortunately, there was a shortcut to making a relevant 
estimate. The shortcut was to calculate the average change in Tier 1 between 1996 and 2000. 
The average change over four years of the 40 or so institutions in Tier 1 turned out to be 
1.94 positions. 

A finding of 1.94 positions is interesting because it represents only 5 percent of the range 
between ranks of 1 and 40. Inspection of the rankings from year to year showed that they 
were very tightly confined, going up or down at most only a few places. The greatest upward 
movements, from the 1996 to the 2000 rankings, were by Davidson College and Macalester 
College, each of which moved up 10 places. Bard College moved up 7 positions. 

If movement happened similarly within the other three (unranked) tiers, we may expect that 
movement between tiers should be fairly rare. Indeed, it was. For our institution, for 
example, the most relevant movement would be from Tier 4 to Tier 3. There were 40 
institutions that were in Tier 4 who had, theoretically, the opportunity to move up between 
the 1996 and 2000 rankings. There were seven that that did move up: The College of St. 
Benedict, Guilford College, Houghton College, Moravian College, Siena College, Western 
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Maryland College, and Westminster College. This number was equivalent to 17.5 percent or 
about 1 in 6. 

If membership within the Best Colleges list is conservative, changing only rarely, and a 
relatively small number rise in rank, it is possible that the overall movement pattern within 
the list can be best described as “ zero sum.” That is, upward movement of some colleges 
is balanced and compensated for by the downward movement of other colleges. For 
example, while the seven colleges moved from Tier 4 to Tier 3, five colleges made the 
reverse trip from Tier 3 to Tier 4: Albright, Bennington, Chatham, Morehouse, and 
Westmont. 

Again there were 40 institutions in Tier 3 that had, theoretically, the opportunity to move up 
between 1996 and 2000. There were six that moved up: Austin College, Hollins College, 

Mills College, Millsaps College, St. Mary’s College of Maryland, and Sweetbriar College. 
Further, there were eight colleges that moved down from Tier 2 to Tier 3. They were: Albion 
College, Hendrich College, Lake Forest College, Lewis and Clark College, Ohio Wesleyan 
University, St. John’s College (N.M.) and Wittenbei^ University. 

Another related question is how much the institutions that rose (or fell) in their Tier 
assignment actually changed in rank. Again, the existing data do not permit an easy answer 
to that question. However, we may turn to Tier 1 once more to find the answer to an 
analogous question at that level. Suppose we create an artificial division within Tier 1 
between institutions with a rank greater than 20 (i.e. 21-40) and those with a rank less than 
or equal to 20 (i.e. 1-20). Let us call them Tier lA (most premier) and Tier IB (less premier). 
If we look for movement between the two new groups, we can measure the change in rank 
of those colleges that moved in either direction. It turns out there were only two institutions 
that rose in rank. They were Davidson College and Hamilton College. Davidson moved 
from ranks 21 to 11 and Hamilton from 23 to 18. There were two other close calls. 
Macalester moved from 34 to 24, thus approaching the fictitious Tier lA status. Mt. 

Holyoke moved from 19 to 16, all within Tier lA. That college was close to qualifying only 
in the sense that its starting point was close to the border between the two sub-tiers. How 
many colleges made the reverse movement from Tier lA to Tier IB? One institution. Bates 
College, moved from rank 18 in 1996 to rank 23 in 2000. A close call was Colgate, which 
moved from 17 to 18 during that period, but had fallen to 21 in 1999. 



Implications 



The information presented in this paper provides one important source of guidance for 
institutions wishing to improve their rankings. In general, there is sufficient stability within 
the rankings to make one take a somewhat skeptical view towards efforts to improve an 
institution’s rankings. At the same time, the same finding bodes well for those institutions 
that happen to be content with their rankings because they are as high as they can be or as 
high as college representatives expect them to become. Such institutions can afford perhaps 
to relax their efforts in regard to raising or maintaining their high status as far as these 
rankings are concerned. However, this general conclusion, while defensible in light of the 
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resialts, shovdd not be construed as being discouraging to those institutions that would like to 
improve their rankings. Neither should the results be construed as offering an excuse for 
complacency among those institutions that happen to be content with the rankings that they 
alreacfy^ possess. 

Before coming to those conclusions, several cautions should be observed. First, it is 
exceedingly important for an institution to determine its approximate rank within the tier to 
which it has been assigned. (For tier 1 institutions this information is provided and easily 
retrieved.) Without such information, it is impossible to determine whether efforts to 
improve are likely to bear fruit within less than 10 years. Equally true, it is impossible to 
know without estimating those rankings whether the institution is at risk of moving 
backward in tier assignment or falling off of the list altogether. 

A second caution is that institutions would do well to reflect upon what they would do if the 
rankings did not exist at all. They should consider whether the efforts expended to improve 
or maintain rank or tier assignment are substantially the same as the efforts that would be 
expended if U. S. News and World Report were not in this business. If the answer to that 
question is that the efforts would be substantially the same, then it might make better sense 
for the institution to use scarce resources pursuing the goal of improving its rank. Those 
efforts could be defended since they merely duplicate the efforts that would be expended if 
there were no rankings to drive them. 
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